Slow Emergence of Cooperation 
for Win-Stay Lose-Shift on Trees 



Elchanan Mossel Sebastien Roch 

Department of Statistics 
University of California, Berkeley 
Berkeley, CA 94720-3860 

{mossel, srochjgstat .berkeley . edu 
February 8, 2008 



Abstract 

We consider a group of agents on a graph who repeatedly play the prisoner's dilemma game against 
their neighbors. The players adapt their actions to the past behavior of their opponents by applying 
the win-stay lose-shift strategy. On a finite connected graph, it is easy to see that the system learns to 
cooperate by converging to the all-cooperate state in a finite time. We analyze the rate of convergence in 
terms of the size and structure of the graph. [Dyer et al., 2002] showed that the system converges rapidly 
on the cycle, but that it takes a time exponential in the size of the graph to converge to cooperation on 
the complete graph. We show that the emergence of cooperation is exponentially slow in some expander 
graphs. More surprisingly, we show that it is also exponentially slow in bounded-degree trees, where 
many other dynamics are known to converge rapidly. 



Keywords: Games on Graphs, Learning, Prisoner's Dilemma Game, Win-Stay Lose-Shift, Oriented Perco- 
lation, Emergence of Cooperation. 



1 Introduction 



We consider a group of agents arranged on the nodes of a graph who repeatedly play the prisoner's dilemma 
game against their immediate neighbors. The players adapt their actions to the past behavior of their oppo- 
nents by applying the so-called win-stay lose-shift strategy [NS93 1 which, as the name suggests, consists in 
changing strategy whenever the payoff is deemed unsatisfactory. This model has been studied in the artifi- 
cial intelligence literature IKi95l as a simple example of "co-learning" IST93II5T97I . On a finite connected 
graph, it turns out that the system converges to the all-cooperate state — the globally optimal state — in finite 
time. In this respect, this instance of the iterated prisoner's dilemma (IPD) game on a graph provides an 
interesting example of a system learning to behave optimally by a mechanism that involves each agent ap- 
plying independently a simple strategy — or rule of thumb — which takes into account only the latest actions 
of its immediate neighbors. For related work, see fPT.98 1 and references therein. See also IAx84l for the 
evolutionary perspective. 

In order to understand how persistent this "emergence of cooperation" phenomenon is, it is crucial to 
analyze the rate of convergence to the all-cooperate state. Where the convergence is rapid, one would expect 
to observe the optimal, cooperation state in a practical system based on similar dynamics. On the other hand, 
where the convergence is slow, one would rather expect that such a system would stagnate in a suboptimal, 
metastable state where a nonnegligible fraction of agents defect. Rates of convergence for IPD were studied 
in IKi95llDG^()2l where the structure of the graph was shown to be a determining factor. 

In this paper, we show that IPD exhibits an exponentially slow convergence to cooperation on expander 
graphs and bounded-degree trees. Our result for bounded-degree trees is somewhat surprising. In particular, 
it should be compared to the behavior of global reversible dynamics on trees llBK + 05l where the conver- 
gence is always rapid. Note however that this slow convergence on trees is not unprecedented. Notably, 
the contact process, a common model of infection, is slow to converge on trees when the infection rate is 
large. See e.g. ILi99 1 and references therein. In fact, our proof suggests that IPD behaves very much like the 
contact process. Nevertheless, the analysis of non-reversible particle systems has been an open challenge 
in the last two decades and we hope that the results obtained here can shed some more light on how such 
systems can be tackled. 

The proof of slow convergence we give here combines several ideas. The main idea is to look at the 
process at the right space-time scaling. This approach, commonly used in probability (e.g. in the analysis 
of interacting particle systems 10851 ). allows us to analyze the rough behavior of IPD — defection survives 
for long periods of time in zones that are densely populated by defectors. The main technical difficulty is 
to control the dependencies between different regions and different times. Then the process is compared to 
a directed percolation process (where the directed axis corresponds to the time axis in the original process). 
Using contour arguments we show that the directed percolation process survives for an exponential time. 
See IDu84l for background on directed percolation. 

1.1 Definitions and Previous Work 

Recall that the prisoner's dilemma game (PD) is a bimatrix game with the following payoff matrix for the 
row player (and similarly for the column player): 



where T > R > P > S and 2R > T + S. The first row (column) corresponds to the cooperate action 
and the second row (column) corresponds to the defect action. The global — or Pareto — optimum is for both 
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agents to cooperate. However, for any given action of the column player, it is always in the row player's 
advantage to defect (and similarly for the column player). 

For an agent playing PD, a simple way to adapt to her opponent's behavior is the so-called Win-Stay 
Lose-Shift strategy (WSLS) INS93I . also known as the Pavlov strategy IKi95IIST97l . This works as follows. 
Every time the game is played, if the agent's payoff is one of the two smaller payoffs, i.e. P or S, then she 
switches her action in anticipation for the next round of play, otherwise she keeps the same action. 

We now consider a repeated graphical version of PD which we will refer to as IPD. Let G = (V, E) be a 
finite graph with n = \ V\. Each node, v, is an agent to which we associate an action A t (v) G {C, D} at time 
t G R+. (As will become clear in later sections, it is easier to consider the continuous-time version of this 
problem.) Here C stands for cooperation while D stands for defection. The initial state is Aq(v) = D for all 
v G V. The agents repeatedly play PD against their immediate neighbors in the graph through the following 
mechanism. Each edge e G E has an exponential clock, i.e. we associate to each edge an independent 
Poisson process {Tj(e)}j>i where all inter-arrival times Tj+i(e) — Tj(e) are independent Exp(l) (with the 
convention To = 0). Every time a clock rings, say at edge e = (u, v), the endpoint agents u and v play one 
round of PD using their respective actions A t (u) and A t (v), assuming the clock rings at time t. Then the 
two agents update their state using WSLS. In other words, if a clock rings on edge e = (u, v) at time t, we 
witness the following transition for (A t (u), A t (v)) 

(C,C) -> (C,C) 
(C,D) - (D, D) 
(D,C) - (D,D) 
(D,D) - (C,C). 

This defines a stochastic process for the state of the system A t = (At(v)) vtE y with initial state the 
all-defect state, Ao = D = (D, . . . , D). It is clear that, given the above allowed transitions, the system 
has a unique fixed point, the all-cooperate state C = (C, . . . , C). In particular, if G is a finite connected 
graph with n > 2, we have a.s. A t — ► C as t — > +00. The question of interest is: how long does it take to 
reach C on a given graph. It was shown by lDG + 02l — and previously conjectured in [Ki95 | — that the time 
to the emergence of cooperation depends crucially on the structure of the graph. Let Tc be the stopping 
time at which A t reaches C for the first time. Below, with high probability (w.h.p.) means with probability 
1 — l/poly(n) where poly(n) increases polynomially with n. In llDG + 02l . the following two results are 
proved. 

Theorem 1 ( |DG + 2|) Let G be a cycle on n vertices. Then w.h.p. Tc = 0(n log n). 
Theorem 2 ( lDG + 02l ) Let G be the complete graph on n vertices. Then w.hp. Tc = Q((l.l) n ). 

1.2 Our Results 

Given the previous theorems, it is natural to conjecture that the time to the emergence of cooperation is 
governed by the connectivity of the graph: a high connectivity, as in the complete graph, leads to slow 
convergence, while a low connectivity, as in the cycle, leads to fast convergence. Surprisingly, we refute this 
intuition with our main result. 

Theorem 3 There is a constant d so that for all n there is a d-regular tree with n vertices for which w.h.p. 
Tc = 0,(p n )for some p > 1 depending only on d. 
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Figure 1: Caterpillar of degree 7. 



To prove this result, we study IPD on "linear trees." The main technical ingredient is a coupling with oriented 
percolation. The proof of this theorem is given in Section|2] 

Although the connectivity conjecture turns out to be wrong in general, the following theorem, an ex- 
tension of the complete graph result of lDG + 02l . shows that the intuition is partly correct in one direc- 
tion. Let G be a graph with n vertices. Let a, (3 be two increasing functions of n such that for all n, 

< a(n) < (3{n) < n. Define the (a, /3)-expansion constant p a ^(G) of G as 

p a ,p(G) = min| J^gp :UCV, «(n) < \U\ < fi(n)\ , 

where E(U, U c ) is the set of edges between U and U c , vol(C/) is the sum of the degrees of the nodes in U, 
and \X\ is the cardinality of X. 

Theorem 4 Let e > 0. Let a, (5 be two increasing functions ofn such that for all n, < a(n) < j3{n) < n. 
Let G be a graph with n vertices such that p a ,(3{G) > 1/2 + e. Then there is a constant a > 1 (depending 
only on e) such that w.h.p. Tc = fJ(a^ n ) _a ( n )) (for n large enough). In particular, if a, (3 are linear in n, 
the emergence of cooperation is exponentially slow. 

This follows from a martingale argument similar to that used in which is detailed in Section|3] Note 

that in Theorem|4j in order to obtain slow convergence, it suffices to have large expansion for relatively small 
sets. In particular, the theorem applies to expander graphs such as random regular graphs IKa95lFKS89l . 

2 Win-Stay Lose-Shift on Trees 

In this section, we analyze IPD on caterpillar trees of degree d. We define an (n,d)-caterpillar, denoted Sj, 
to be a tree with the following property: the subtree induced by the internal nodes is a path containing n 
nodes all of which have degree d. See Figure Q Our main result, Theorem|3j is that cooperation is slow to 
emerge on caterpillars. The proof of Theorem [^follows from a series of stochastic domination arguments. 
We now briefly outline the main steps of the proof. 

1 . Star Dynamics via Biased Random Walk. The first step is to analyze the behavior of a single star. 
The main point here is that it takes the star with d leaves an exponential number of steps (in d) to move 
from the all-defect state to the all-cooperate state. This is proved by comparing the process to a biased 
random walk. This comparison also shows that a star can go from a few defectors to linearly many in 
poly(d) time with constant probability, and that a small linear fraction of defectors grows with high 
probability within poly(d) steps. Moreover, these claims can be established even if one allows two of 
the nodes of the stars to have arbitrary values. 

2. Space-Time Scaling. We think of a star as defecting if at least d/4 of its leaves defect. Then, we 
consider triplets of adjacent stars and say that a triplet is defecting if at least one of its extremal stars is 
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defecting. (We actually work with triplets of stars rather than pairs to help control dependencies.) We 
scale time by looking at the process every poly(d) steps. The random walk argument of the previous 
point allows to show that defecting stars have a high probability — at least (1 — exp(— 0(d))) — of 
remaining defectors after the poly(d) time window. Moreover, a defecting star has a l/poly(d) 
probability of "infecting" neighboring stars during that time. By iterating these observations poly(d) 
times — yet another time scaling — we show that a defecting triplet has a probability 1 — exp(— 0(d)) 
of "infecting" a neighboring triplet. (Neighboring triplets are actually intersecting.) 

3. Percolation. We may now look at the space-time diagram of defecting triplets and show that it 
dominates a directed percolation with probability 1 — exp(0(— d)) for edges to be open. The time 
axis of the original process corresponds to the direction of propagation in the percolation process. 
Finally, a contour argument allows to conclude that this percolation survives for a time which is 
exponential in n, thus proving that the convergence time of IPD on the caterpillar is itself exponential 
in n. 

2.1 Star 

Let G = S^. This graph is made of n copies of (i.e. stars of degree d). Let G' be any star in G. Denote 
the root and the leaves 1,2, ... ,d. A crucial property of stars is that cooperation is slow to emerge on 
them. This follows from our next result. We single out nodes 1 and 2, which are defined to be the two nodes 
that O shares with its neighboring stars. (In the case of extremal stars, we just pick an arbitrary node in 
addition to the node shared with the next star.) We call 1 and 2 the external vertices. We use the following 
notation: a A b = min{a, b}. 

Lemma 1 (Dynamics on Stars) Consider the IPD chain { A t }t>o on G = w ' tn d > 15. Let G' be an 

arbitrary star in G with nodes denoted 0, . . . , d (0 being the root, and 1 and 2 being the external vertices). 
Let M' be a positive integer and go,9i,92 be three increasing functions of d with 52(d) = d/3 — 2 and 
go,gi satisfying 1 < go(d) < gi(d) < 52(d) for all d. Let the initial configuration be as follows. On G', 
nodes 3 through d — g\ are C and nodes d — g\ + 1 through d are D. On all other nodes, including the root 
and external vertices of G ', the initial state is arbitrary. Define 

A D = |{*e{3,...,d} : A(i)=D}\. 

Let T g be the first time Ad = g. Let A2 = 52 — gi, Ai = g\ — go, p = a/9/8, and p = goM 1 . Then, we 
have 

P[T g2 > (T go A M')] < 2~ Al + ^v^/2(y2) A2 + 2"^ /2 . (1) 

Moreover, this bound applies simultaneously on all stars independently from each other (possibly with dif- 
ferent choices ofg's). 

Proof: For this argument, we restrict ourselves to what happens on G' and do not refer to any event involving 
the rest of G. We call a leaf edge with leaf state D a D-edge, and similarly for C. The behavior of Nrj 
depends on the state at the root of Q . When A(0) = C, nothing happens until a D-edge is picked at which 
time A(0) becomes D itself. On the other hand, when ^4(0) = D, either a C-edge is chosen in which case 
iVo may go up by 1 (or stay the same if 1 or 2 is picked), or a D-edge is chosen in which case A^d may go 
down by 1 (or stay the same if 1 or 2 is picked) and A(0) becomes C. Ignore the updates where nothing 
changes, i.e. when an edge (C, C) is chosen. In any configuration satisfying Ad > go, there are at least 
go edges whose updates change the configuration. Let Q the number of such updates in time A f. Then it 
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follows that Q is larger than a Poisson with mean p = goM'. From the moment generating function of the 
Poisson distribution (see e.g. [Du96|), we have the following 



>[Q < VMl = IP[e- Q > e^] < -^J < ^ 



Assume the event {Q > ^/p} holds. Also, note that at most one out of 2 steps have A(0) = C. (Remember 
that we ignore (C, C) updates.) Ignore the times with A(0) = C as well, what remains is an asymmetric 
random walk (or rather a birth-and-death chain) which does at least y/p/2 steps before time M' . To bound 
the probability that No goes up or down, we use the fact that the chain starts with g\ D's and is stopped 
when it reaches either go or 92 D's. By assumption, the probability that goes up when ^4(0) = D is at 
least (d — 2 — 92) /d. Consider the walk {Sk}k>o on N started at Sq = g\ which goes up with probability 
p = (d — 2 — $2)/^ = 2/3 and goes down with probability 1 — p = 1/3. Let T' g be the time at which Sk 
reaches g. For convenience, we assume that the process {Sk}k>o is defined on all of Z (even though outside 
the interval [30 ? #2] the bounds used are not valid). Then, 

F[T g2 > (T go A M') \Q>^p]< F[T' g2 > (T; A v^Z/2)] < F[T' g2 > T'J + F[T> 2 > Jp/2\. 

By standard martingale results (see e.g. IDu9 61). we have 

0(A 2 )-^(O) 



\T' > T' } 
l 92 — ^goi 



where 



So, 



We also have 



FIT' > T 
L .92 — a 



1-p 



1 - 2" 



-Ai 



92 — goi _ 2 _A 2 



< 2" 



np 1 



1 - \J\ - 4p(l - p)p 2 



The choice p = y/9/8 gives 



E[p T 92] = ( ^2 



By Markov's inequality, 



F[T' > y/ji/2] = F[p T 32 > < p -v^/2 f^2 



Finally, putting everything together, we get Q. 

The independence of the bound at each star in G comes from the fact that we use only events involving 
leaf edges of G' . 



The following corollary corresponds to the case where a star has initially only a few D's. The result 
below implies that after M' = poly(d) steps the star has 0(d) D's with positive probability. 
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Figure 2: Reduction to percolation. 



Corollary 1 (Defection Spreads on Stars) In the setup ofLemma\I] let go(d) = 2, g±(d) = 3 and g2(d) = 
d/3 — 2. Then, for M' = u>(d 2 ) and d (constant) large enough, we have 

P[T ff2 >(r so AM')]<| 

The following corollary implies that a star with 0(d) D's still has 0(d) D's after poly(d) steps, with 
high probability. 

Corollary 2 (Defection Survives on Stars) In the setup of Lemma\l\ let M' = +oo, 52(d) = d/3 — 2, 
gi(d) = d/3 — 3, and go(d) = d/A — 3. Then, 

m 2 > T go ] < 2~ d / u . 

The following corollary implies that a star with d/A D's reaches d/3 D's after poly(d) steps, with high 
probability. 

Corollary 3 (Defection Boosting on Stars) Let r be a positive integer, not depending on d. In the setup of 
Lemma{l] let 52(d) = d/3 — 2, g\(d) = d/A — 2 — r, and go(d) = d/5 — 2 — r. Then, for M 1 = u>(d 2 ) and 
d large enough, we have 

P[r ff2 > {T go A M')] < 32~ d/2 ° < 2~ d/21 . 

2.2 Star Triplets 

The next step in the proof of Theorem |3] is to make the connection between IPD and oriented percolation. 
Here we show how a triplet of stars dominates the building block of a percolation lattice. We use the 
following oriented percolation. Consider four adjacent vertices of the regular lattice Z 2 , say voo = (0, 0), 
^01 = (0,1), vio = (1,0) and v\\ = (1,1). Assume the nodes are connected by four directed edges: 
eo = (uoo,Uoi), ei = (fio^n), e i = (foo^n), and e w = (v w ,v i). See Figure|2l Each edge is open 
with respective probability po, p±, poi, and pio. The vertices have a state, denoted respectively sqo, sqi, s\o, 
su, which takes its value in {0, 1}. The state 1 "travels"along the open edges, i.e. if e = (u, v) is an open 
edge and the state at u is 1 then the state at v is also 1. A vertex is in state 1 if and only if it is the terminal 
vertex of an open edge with initial vertex in state 1. We denote this four-node graph Hb- 
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Now consider any triplet of adjacent stars inside C = SJJ- Denote the stars Sj, j = 1,2,3, with 
corresponding edges {e^}j =1 and vertices {vf }f =0 , with the label corresponding to the root. We have 

the correspondence = and dp = e i • We denote this subgraph — which is a copy of §> d — G' . We 
are interested in the number of D's on each star, excluding nodes 0, 1, and 2 of each star, which we denote 

N t = (iV t (1) ,iV t (2) ,jvf ). 

The detailed behavior of N 4 is rather intricate. We simplify the process by projecting it to a smaller 
space. Let 

f 1, ]£N>d/4-2, 
\ 0, if otherwise. 

Consider the random vector 

s = ($qo, Soi^o, Sn) = (<rd[N^\a d [N^\a d [N^],a d [N^\ 



for some M > 0. The following lemma shows that for an appropriate choice of M, po, p\, poi, and pio, the 
vector s stochastically dominates 

s = (soo,soi,sio,sn), 
defined by the percolation above (with soo = «oo an d sio = 5io). 

Lemma 2 (Connection to Percolation) Consider the IPD chain {A t }t>o on G = §>2 with d > 15. Let G' 
be an arbitrary triplet of adjacent stars in G. Let M = d 6 , po = p\ = 1 — 2~ d / 30 , and poi = Pio = d~ 10 . 
Then, for any initial configuration and Soo, s io such that sqq = sqo and sio = Siq, we have that (5qi, sh) 
stochastically dominates (sol, s n) fo r d (constant) large enough. Moreover, the domination holds for any 
number of ( edge- jnonintersecting triplets simultaneously independently from each other. 

Proof: The argument ignores any event outside G' . We consider three cases. 

1) Case 5oo = sio = 0. In that case, we have soi = su = 0, which is of course dominated by 5oi, Sn. 

2) Case 5oo = sio = !• We use corollaries|2land|3j which we apply to stars 1 and 3 independently. Consider 
star 1. We first go through a "boosting" phase where we let iV^ 1 ) drift from d/4 — 2 to d/3 — 2. Then we 
compute the probability that A^ 1 ) stays above d/4 — 2 for the remaining time. 

Phase 1. For the boosting phase, we apply Corollary [5] The probability of remaining below d/3 — 2 is at 
most T d l 2X . 

Phase 2. The time remaining after boosting is of course at most M. In time M, there is a Poisson number 
of steps, say Q', with mean dM (including the steps where nothing happens). From the moment generating 
function of the Poisson distribution (see e.g. IDu96l ). we have the following 

TffloQ'l „dM(e-l) 

F[Q' > d 2 M 2 ] = P[ e <7 > e d2M2 ] <%l< e -^r- < 2~^ 2 I\ 

Assuming d/3 — 2 was reached and that there remains at most d 2 M 2 discrete steps, we get that there are 
at most d 2 M 2 crossings of the interval [d/4 — 3, d/3 — 2] by the process iV^ 1 ). By Corollary |2j every time 
= d/3 — 3, there is a probability of at least 1 — 2~ d l 12 of coming back to d/3 — 2 before hitting 
d/4 — 3. The probability that any of d 2 M 2 attempts at crossing [d/4 — 3, d/3 — 2] succeeds is at most at 
most d 2 M 2 2~ d/12 which implies 

P[5io = 0] < d 2 M 2 2~ d l 12 + 2~ d2M2 ' 2 + 2~ d l 21 < 2~ d l 22 , 

for d large enough. Stochastic domination of the oriented percolation follows directly. 
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3) Case sqq = 1, s*io = 0. (The symmetric case is analyzed similarly.) We divide the time window in two 
phases. For the first phase, we compute the probability that defection "spreads" from star 1 to star 3. For the 
second phase, we compute the probability that stars 1 and 3 remain in or reach state 1 respectively. 

Phase 1. It is easy to see that, in any initial configuration satisfying s*oo = L = 0, six steps (or less) 
suffice to reach a configuration with > 3. The probability that the first six steps taken by IPD satisfy 
this property — call that event B — is at least 1/d 6 . Let Q" be the number of steps until time M/2. Then, 

W < 5] < 2- M '\ 

by a calculation similar to that in Lemmaffl 

Phase 2. We condition on {Q" > 6}. Consider first star 1. Whether or not B is realized, at the beginning of 
Phase 2, we have > d/4 — 8. We are back in the situation of Case 2), except that the time left is only at 
least M /2. By the same calculation, we obtain that the probability that Sio is is at most 2~ d l 22 for d large 
enough. Consider now star 3. Let Q'" be the number of discrete steps left on star 3. The time remaining 
is at least M/2. It follows from Corollary Q that reaches d/3 — 2 before the end of the time window 
with probability at least 1/3 for d large enough. Once d/3 — 2 is reached, we are back to Phase 2 of Case 
2). It follows that on {Q" > 6} the probability that §n = 1 is at least d~ 6 /4. Note that on {Q" > 6}, the 
bounds on star 1 and 3 are independent. It is then easy to check that stochastic domination of the oriented 
percolation holds. 

■ 

We further simplify the chain by stacking up the construction in the previous lemma and projecting 
once more to a smaller space. For this, we consider a different percolation model on Z 2 . See Figure [3] 
Let H' B be the directed graph made of three nodes v' 10 = (1, 0), v' 01 = (0, 1), v' 21 = (2, 1) with two edges 
e i = (^io'^oi)' e 2 = ( v xa-> v 2i)- The edges are open with probability p[, p 2 respectively. The nodes have 
state s' 10 , s' 01 , s 2l respectively with value in {0, 1}. The percolation works as before with state 1 "traveling" 
along open edges. 

Consider again IPD on an arbitrary triplet of stars G 1 of G. Redefine the vector s by taking instead 

s = (Soo,5oi,Sio,Sil) = (a d [N^%o d [N^lo d [Nf\o d [Nf^ , 

for some JgN and M as in Lemma[2] We use the following notation: a V b = max{a, b}. 

Lemma 3 (Towers) Consider the IPD chain { A t }f>o on G = with d > 15. Let G' be an arbitrary 
triplet of adjacent stars in G. Let M = d 6 , I = d 100 , and p\ = p' 2 = 1 — 2~ d ^ 100 . Then, for any initial 
configuration and s' 10 such that s' 10 = §oo V sio, we have that (sqi, in) stochastically dominates (s' 01 , s' 21 ) 
for d (constant) large enough. Moreover, the domination holds for any number of (edge-)nonintersecting 
triplets simultaneously independently from each other. 

Proof: The argument ignores any event outside G' . The proof works by stacking up / copies of Hb 
and applying Lemma 13 Consider again Z 2 . We define a I-tower, denoted H B , to be the graph on nodes 
{ v o,i = (0, i), = (1, i)}i=o where each set of four nodes of the form {uo,i, vo,i+i, v i,i+i} induces 
a copy of Hb with the same values of Po,pi,Pw,Poi as in Lemma[2] The node states are denoted {so,i = 
(0,i),s lti = (l,i)}l =0 . By applying repeatedly Lemma|2l we get that, if (s 00 ,s 10 ) = (so,o,si,o), then 
(soi; sn) stochastically dominates (soj, sij), so it suffices to show that the latter dominates (s' 10 , s' 21 ). 

The case s'iq — is trivial. So assume — 1. Then, the subcase sqq A s\q — 1 dominates the subcase 
sqo A sio = so it suffices to consider the latter. Without loss of generality, let s*oo = 1 and s"io = 0. The 
probability that at least one upwards edge in H B is closed is at most 

o T { -d/m\ < 9 -d/3i 
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G> Hi H' E 

Figure 3: Further reduction. 



for d large enough. The probability that no up-right edge is open is at most 

( 1 - —V < 2~ d/31 

for d large enough. Therefore, 

P[s ,J = so,/ = 1] > l-2~ d / 32 , 

for d large enough. But note that 

P[ a ' 01 = s ' 21 = 0] = ( 2 - d / 100 ) 2 = 2" d / 50 > 2~ d / 32 . 

So we have domination. 
■ 

2.3 Oriented Percolation 

We conclude the proof of Theorem [5] by showing that the IPD chain at intervals of time IM dominates a 
standard percolation model and that in turn the latter model percolates at an exponential distance from its 
bottom nodes. 

For convenience, assume n is of the form 

n = 2n + 1, 

for some positive integer n'. (The reason for this choice will be clear below. See also Figure |5]) Consider 
the following sublattice of Z 2 , 

V = {{i,j) €Z 2 : 1 < % < ri, < j < T, % + j is even}, 



9 



Figure 4: A section of the oriented percolation lattice. 



where T is a positive integer that will be fixed below. Consider the directed graph G-p = (Vp, E-p) with 
node set Vp = {vi,j}uj)£.-p and edge set 

E v = {(vi t j,Vi + ij + i),(vi t j,Vi-ij+i)}(ij)ep. 

See Figure 0] for an illustration. Each edge has probability p' of being open where p' is set below. We 
consider the percolation process on G-p and denote the states sip = {s^ j}^j^p. 

Let { A t }t>o be the IPD chain on and denote the number of D's on star i at time t, excluding 
the external nodes. We consider the following projection of {At}t>o- Let 

= 4(i - 1) + t{j i s odd}, 

and let s = where 



Ar(At(*j')-l) 
ly jIM 



where / and M are as in Lemma|5] See Figure|5] We show first that s dominates s'. 

Lemma 4 (Domination of Oriented Percolation) Consider the IPD chain { A t } t >o on G = 8^ w ' tn d > 
15. Let M = d 6 , I = d 100 , and p' = 1 - 2~ d / 100 . Let A = D (the all-B state) and let s' i0 = I for all even 
i's. Then, we have that s stochastically dominates s' for d (constant) large enough. 

Proof: This actually follows immediately from Lemma|3] 
■ 

Finally, the next lemma concludes the proof of Theorem |3] 

Lemma 5 (Crossing) Let s' be defined as above with p' = 1 — 2 _d / 100 and let s' i0 = I for all even i's. Let 
T = 2( d / 200 °) n . Assume that n = 2ri + 1 and that T is even. Then 

r[s' iT = 0, Vi G {2, 4, . . . , ri - 1}] < 2" W 100 °) n , 
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for d (constant) large enough. 

Proof: We use a standard duality argument. For more details, see 8Du841. First we modify the percolation 
lattice Gp, which we now call the primal lattice and still denote Gp. To each edge, we add another edge, 
reversed, with associated probability of being open 0. We now define the dual lattice. Let 

V = eZ 2 : 1 < i < n', < j < T, i + j is odd}. 

Consider the directed graph G-p = (Vp,Ep) with node set Vp = {vij}uj) e x> an d edge set 

Superimpose Gp> on top of Gp and notice that to each edge of Gp corresponds an edge of G-p which is 
rotated 90° clockwise. See Figure|6l We couple the two lattices so that an edge in Gp is closed if and only if 
the corresponding edge in Gp is open. It is not hard to see that there is an open path from level to level T 
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in G-p if and only if there is no open path from the right boundary to the left boundary in Gx>- So it remains 
to compute an upper bound on the latter probability. Fix any two boundary nodes in Gx>, say vi = v\. q and 
v r = v n r £ for some r], £. The number of paths of length L between v r and v\ is at most 3 L . Each such path 
makes n' — 1 more moves to the left than to the right. In particular, the number of moves to the left is at 
least L/2. Moreover, each edge going to the left has a probability 1 — p' of being open. So the probability 
that there is a path between v r and vi (which we denote v r ^ v{) is at most 

p[,v^,,i< £ 3^ ( i-po^< ( f_ 32 J /2 ; . 

L=n'— 1 

for d large enough. There are at most T 2 pairs of boundary nodes so by the union bound 

P[< T = 0, Vi e {2, 4, . . . , n> - 1}] < T 2 ( ^ 32 _j 200 < 2~W™^, 
for d large enough. 



3 Win-Stay Lose-Shift on Graphs with Large Expansion 



For this section, we consider the discrete-time version of the chain. That is, at every time step, we pick one 
edge uniformly at random and update the actions at the endpoints of that edge. Equivalently, we look at the 
discrete-time chain embedded in { A t }t>o by stopping the chain every time a clock rings. Also, since we 
are looking for a lower bound on Tc, we can speed up the chain by picking only those edges with at least 
one D endpoint. Denote the discrete-time sped-up chain {B/J^g^. 

The proof of Theorem |4] is based on the following geometric observation. Let Uk be the set of nodes 
defecting at time k and denote iVfc = \Uk\- At the next update, goes down by 2 if we pick an edge 
"inside" and it goes up by 1 if we pick an edge on the "boundary" of Uj.. Therefore, if the boundary of 
Uk is more than twice as big as the inside of Uk, on average the chain moves away from the fixed point C. 

Proof of Theorem HJ Let U C V with a(n) < \U\ < 0(n). Note first that p a ,p{G) > 1/2 + e implies 



\E(U,U C )\ > l- + ejvol(U). 

Let e' > such that 2 - e' = (1/2 + e)" 1 . Then 

\E(U, U c )\ + 2\E(U, U)\ = vol(U) < (2 - e')\E(U, U c )\, 

which implies 

2\E(U,U)\ < {1- e')\E(U,U c )\. 
Therefore there is an e" > such that if a(n) < JV& < j3{n), then 



fc+i 



Nk + 1, with probability at least | + e" , 
Nk — 2, with probability at most A — e" . 



Let 



1 (2 



2 V 3 



-1 



n l/3 



> 1. 
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l + e") cr x +(l-e")a 2 < 



It is easy to check that 

(3 1 ~ j ~ 1 V 3 

Therefore, 

W{N k )=a n - N \ 

is a bounded nonnegative supermartingale on {a(n) < < (3(n)}. Using the optional sampling the- 
orem as in |DG + 02| . it follows that the probability of N k crossing the interval [a(n), (3(n)] is less than 
a -(/?(n)-a(n)) f or n enough. The theorem immediately follows. 



4 Concluding Remarks 

The work presented here leads naturally to the following questions: 

1 . Is there a d (constant) such that for all n large enough and for all trees of minimum degree d with n 
nodes, the emergence of cooperation is exponentially slow? 

2. What is a good criterion for fast emergence of cooperation in this setup? Is the line and its — 
appropriately defined — variants the only graphs on which the convergence to all-cooperation is fast? 
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